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Abstract 

This  paper  describes  the  administrative  philosophy  that  currently 
guides  the  (evolutionary)  acquisition  ofU.S.  military  systems.  It  then 
sketches  a  preliminary  mathematical  model  that  allows  study  of  the 
effect  of  various  ways  to  spend  a  fixed  budget  for  Block  b+1  upgrade 
so  as  to  obtain  a  maximum  expected  number  of  fielded  system 
upgrades  that  is  effective  in  the  field.  This  includes  the  option  of 
simply  fielding  more  of  the  previous,  Block  b,  design  units. 
Effectiveness/ “capability’’  growth  is  the  design  objective,  but  testing 
and  fault  removal  provides  for  reliability  growth.  The  model  accounts 
for  various  levels  of  developmental  and  testing  effort  at  various  rates, 
and  for  obsolescence  of  the  previous  (Block  b)  and  forthcoming 
(Block  b+1)  system  versions. 


Keywords:  evolutionary  acquisition;  operational  testing;  reliability  growth 


1  The  Current  (2004-05)  Acquisition  System 

Modern  (circa  ‘04-‘05)  U.S.  military  acquisition  follows  a  top-down  policy  pattern, 
governed  by  a  Joint  Capabilities  Integration  and  Development  System  (JCIDS).  This 
informs  and  supports  decisions  by  the  Joint  Chiefs  of  Staff  (JCS)  and  the 
Joint  Requirements  Oversight  Council,  which  identifies,  assesses,  and  prioritizes  joint 
military  capability  needs.  JCIDS  is  concerned  with  methodology  to  identify  and 
describe  so-called  capability  gaps,  review  capability  improvement  proposals  and 
improve  capability  integration,  define  non-materiel  (e.g.,  Concept  of  Operations 
(CONOPS))  aspects  of  materiel  solutions,  improve  interdepartmental  (Joint) 
coordination,  improve  requirement  generation,  adaptation  and  revision,  and 
documentation,  all  in  a  timely  manner;  timeliness  is  intended  to  be  enhanced  by 
spiral  development  and  evolutionary  acquisition.  Program  progress  occurs  according 
to  milestones:  a  new  mission  requirement  (e.g.,  induced  by  changed  threat 
assessment)  is  described  in  an  Initial  Capability  Document  (ICD)  prior  to 


1 


Milestone  (MS)  A,  at  which  time  T&E  strategy  (a  new  name  for  what  was  formerly 
called  a  first  version  of  the  Test  and  Evaluation  Master  Plan,  or  TEMP),  is  defined, 
and  must  be  approved  at  Department  of  Defense  (DoD)  level.  Then  a  Capability 
Development  Document  (CDD)  is  due  by  MS  B,  and  a  Capability  Production 
Document  (CPD)  at  MS  C.  For  major  programs,  the  program  manager  (PM)  submits  a 
TEMP  that  must  be  approved  by  Director,  Operational  Test  and  Evaluation  (DOT&E) 
officials  for  MSs  B  and  C,  and  to  lead  to  Full  Rate  Production  (FRP),  and  to  fielding. 

1.1  The  TEMP 

A  vital  element  of  the  above  process  is  the  creation,  updating,  and  evolution  of  the 
TEMP.  This  paper  will  discuss  issues  that  should  be  addressed  by  the  TEMP  as  it 
guides  a  presently  more  dynamic  process  than  has  been  in  place  in  the  past.  To  quote 
a  recent  instruction  (DoDI  5000.2  (2003)  E5.4.1):  “The  Test  and  Evaluation  Master 
Plan  shall  describe  planned  developmental,  operational,  and  live  fire  testing  including 
measures  to  evaluate  the  performance  of  the  system  during  these  test  periods;  an 
integrated  test  schedule;  and  the  resource  requirements  to  accomplish  the  planned 
testing.”  No  specific  guidance  is  given  on  how  to  plan  testing  or  how  to  integrate  the 
testing  into  a  schedule  for  spiral  development  or  evolutionary  acquisition.  This 
present  paper  elucidates  features  of  a  useful  and  successful  dynamic  TEMP, 
emphasizing  the  use  of  models  (and  simulation  as  required)  and  data  acquisition  and 
analysis  in  that  process.  Special  emphasis  is  given  here  to  the  use  of  testing  to 
enhance  various  Suitability  issues;  these  include  obtaining  a  system  that  can  be  placed 
satisfactorily  in  field  use,  with  special  attention  to  Reliability,  Availability, 
Maintainability,  and  Safety,  (RAMS).  Also  included  are  transportability, 
interoperability  and  manpower  and  logistic  supportability,  robustness  to  environment, 
(e.g.  weather  and  terrain)  and  training  requirements.  However,  testing  is  first  and 
foremost  invoked  to  demonstrate  improved  and  required  Operational  Effectiveness, 
meaning  the  degree  of  improved  mission  performance  over  a  current  system  version 
(Block,  or  baseline),  where  the  focus  is  effectiveness  against  changing  threats, 
meaning  relative  invulnerability  against  weapons  and  countermeasures  while 
accomplishing  its  mission(s).  Mission  accomplishment  is  fundamental,  and 
quantitative  measures  of  effectiveness  and  suitability  should  be  correctly  defined  and 
constantly  updated.  For  this,  testers  must  rely  on  earlier  data,  e.g.,  from 
developmental  Testing  (DT)/subsystem/component  testing,  which,  in  turn,  will  be 
guided  by  preliminary  modeling  (and  simulation).  It  should  be  demonstrated  early  on 
that  the  entire  system  can  be  manufactured  and  integrated,  and  that  it  is  transportable 
and  sustainable.  Useful  and  cost-effective  Initial  Operational  Test  and  Evaluation 
(IOT&E)  must  depend  on  information  from  earlier  development  and  from  experience 
with  other  systems,  and  the  application  of  scientific  knowledge  and  insight.  Testing 
must  be  designed  and  adequate  to  cope  with  defined  (presently  understood)  missions 
having  a  quantified  need,  definitions  and  measures  of  mission  accomplishment, 
realistic  conditions  for  testing,  and  capability  to  interact  with  other  systems. 
Robustness  and  work-arounds  when  unexpected  events  (system  failures  or  threat 
modification)  occur  are  highly  valuable,  and  should  be  effectively  tested.  Adaptability 
is  an  essential  system-wide  feature. 
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1.2  Testing  and  the  Design  Modification  Process:  Fault  Removal,  Block-Block 
T  ransition/Up-Grading 

The  environment  for  testing  under  the  present  more  static  acquisition  philosophy  has 
changed  to  recognize  system  design  evolution;  c.f.  DoDD  5000.1  (2003),  Daly  et  al. 
(2003),  Seglie  (2002).  If  a  system  has  evolved  to  Block  b,  then  changes 
(improvements!)  leading  to  Block  b+ 1  are  in  principle  being  studied  and  tested  at  the 
prototype  “brass  board”  level;  at  some  point  Block  b+ 1  is  operationally  tested  using 
Block  b  as  baseline,  and  will  be  accepted  when  Block  b+ 1  is  “measurably  better”  than 
the  baseline.  Fault  (resulting  from  inadequate  design,  manufacturing,  transportability, 
maintainability)  removal  can  occur  both  within  Block  b  (even  after  fielding,  but  at 
higher  cost).  A  “real  option”  is  to  examine  several  alternative  technologies  and 
CONOPS,  and  select  the  “measurably  best,”  considering  end-to-end 
cost-effectiveness  and  including  estimates  of  the  timing  of  technological  and 
operational  obsolescence,  likely  caused  by  development  of  countermeasures  by  the 
enemy.  Perhaps  several  alternative  versions  of  Block  b+ 1  can  be  acquired  and  fielded 
for  different  mission  types,  and  to  forestall  countermeasures.  The  process  of  balancing 
the  development  process:  the  Test,  Analyze,  Fix,  Test  (TAFT)  action  sequence, 
followed  by  field  Operational  Test  and  Evaluation  (OT&E)  is  designed  to  yield  useful 
and  trustworthy  improvement  in  timely  fashion  (e.g.,  before  Block  b  penalizing 
attrition  or  obsolescence).  A  central  problem  is  to  address  both  the  “effectiveness,  or 
capability  gap  closure”  problem,  and,  importantly,  also  to  address  the  “reliability 
growth”  problem  realistically  and  recognizing  end-to-end  system  requirements, 
important  elements  being  that  the  new  (sub)system  version  interoperates  effectively 
with  other  system  elements  under  the  entire  range  of  mission  alternatives  envisioned. 

1.3  Models  and  Analysis  to  Expedite  Cost-Effective  Transition  Between  Blocks 
(and  Within  Blocks) 

We  outline  and  propose  rational  attacks  on  the  basic  decision  problem:  when  and  how 
to  allocate  resources  (basically,  funding,  but  also  previous  experience)  to  Block  b  to 
b+ 1  transition.  The  particular  models  described  in  this  paper  are  simple  and 
preliminary,  serving  to  illustrate  the  basic  issues  encountered  when  a  sequence  of 
block  upgrades  is  the  evolutionary  acquisition  strategy.  More  details  are  given 
elsewhere.  Of  course,  each  system  requires  its  individual  understanding  and 
characterization — one  that  captures  essential  physical/operational  features,  but  does 
not  dwell  on  minutia. 

At  any  time  point/date  there  will  be  a  certain  number,  B(b;  t),  of  surviving  and  useable 
copies  of  Block  b.  One  option  is  to  continue  with  that  number,  recognizing  that 
attrition  and  obsolescence  will  reduce  that  Block  b’s  mission  effectiveness,  even  if  the 
Block  b  stock  (e.g.,  number  of  current  design  copies)  is  increased  over  time.  The  cost 
of  this  option  will  likely  grow  due  to  probable  increased  need  for  field  repairs,  and 
consequent  minor  modifications  and  force  enhancements.  Also,  effectiveness  and 
reliability  (actually  RAMS)  may  well  diminish.  We  propose  to  construct  and  analyze 
stochastic  models  for  this  option.  An  alternative  option  is  to  introduce  Block  b+ 1  to 
field  use,  after  the  entire  TAFT  (6+1)  process,  conducted  in  both  Developmental 
Testing  (DT)  and  Operational  Testing  (OT)  phases.  We  propose  (further) 
development  of  models  to  guide  timing  of  such  a  transition,  taking  explicit  account  of 
costs  of  design,  DT  and  Operational  (field)  Testing.  An  initial  simplified  version  of 
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the  advisability  and  timing  of  such  a  transition  appears  in  Gaver  et  al.  (2004).  A  more 
comprehensive  model  is  under  construction  that  recognizes  that  many  to  most  systems 
function  roughly  sequentially,  activating  in  succession  various  sub(sub)system 
functional  stages  (consider  a  missile  that  must  first  experience  launch,  then  initial, 
midcourse,  and  final  stage  propulsion  and  guidance,  and  final  activation  of  the 
warhead).  Clearly  end-to-end  testing  must  assure  that  all  such  stages  are  activated  so 
that  design  faults  can  be  revealed — and  “Fixed”;  the  latter  process  is  error-prone,  so 
more  testing  is  required;  indications  of  “how  much  (more)  is  enough”  can  be  assessed 
by  realistic  models,  but  the  crucial  ultimate  test  is  experimental/operational. 
Gaver  et  al.  (2003)  has  discussed  the  properties  of  such  reliability  growth  models  that 
explicitly  recognize  stage-wise  system  architecture  and  more  detailed  and  sequential 
(current-experience-based)  test-stopping  rules.  These  will  be  adapted  to  evaluate  the 
cost-effective  policy  for  Block-to-Block  transition — i.e.,  rationalize  the  pace  of 
evolutionary  acquisition. 

2  A  Simplified  Formal  Model  for  Performance 

The  probability  of  mission  success  for  Block  b+ 1  after  development  and  testing 
depends  on  the  funding  allocated  to  development  and  its  rate  of  expenditure.  The 
development  process  should  not  only  increase  the  effectiveness  of  Block  b+ 1,  but  also 
tends  to  introduce  design  defects  (DDs)  that  testing  can  remove.  We  consider  a 
system  that  can  be  used  once,  such  as  a  missile.  This  model  was  introduced  in 
Gaver  et  al.  (2004). 

Let  Td  ( b  + 1)  be  the  total  (calendar)  time  to  develop,  test,  and  manufacture  Block  b+ 1 ; 
here  Td{b  + 1)  is  a  fixed  “spiral  time,”  which  is  only  one  alternative.  M(Td{b  + 1)) 
represents  the  total  funds  allocated  to  Develop  Block  b+ 1; 
m  (Td  ( b  + 1))  =  M  ( Td  (b  + 1  ))/Td  ( b  + 1)  is  the  average  rate  of  that  expenditure  (ignoring 

feedback  adjustments  to  intermediate  events — successes  and  failures).  A  plausible,  if 
speculative,  model  for  the  performance  (Effectiveness)  of  Block  b+ 1  is  the  fraction  of 
an  objective  mission  success  measure  (“requirement”)  after  expenditures  of 
M(Td  ( b  + 1))  at  rate  m(Td  ( b  + 1)) ,  given  that  any  remaining  DDs,  new  or  old,  do  not 
activate  during  a  mission,  will  be  written  in  the  general  fonn: 

^(h  +  i;rrf(6  +  i);M(rrf(+i)))  =  /(h  +  i)G4h+i;M(rrf(6  +  i));m(rrf(h+i)));  (l) 

where  (a)  p*  (b  + 1)  is  a  goal  level  (“requirement”)  for  mission  success  of  Block  b+ 1 
(realistically  this  could  be  the  maximum  of  various  minimum  acceptable  measures  of 
mission  success,  related  to  a  Pareto  optimum);  while  (b) 
Gd(b  +  l;M(Td(b  +  l))  : \m(Td(b  +  \ )))  is  the  fraction  of  the  maximum,  (a),  achieved  by 

the  development  process,  when  development  budget  is  allocated  to  be  M(Td(b  + 1)) 
expended  at  basic  average  rate  m(Td  (b  + 1))  over  time  period  Td  (b  + 1) . 

We  provide  specific  parametric  illustrations  in  Appendix  1.  It  is  anticipated  that 
Gd  (•)  should  increase  at  a  decreasing  rate  with  Md  (•) ,  but  that  an  inordinately  high 
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rate  of  redesign  and  system  modification  (large  m  (•) )  will  have  a  detrimental  effect 
on  the  Block  improvement.  Note  that  an  optimum  rate  is  impossible  to  determine  for  a 
new/revised  system,  but  studies  of  the  effects  of  successive  development  (“spiral”) 
times,  Td  ,  can  guide  the  choice  of  budgets  and  rates  of  expenditure. 


Let  the  number  of  DDs  introduced  during  initial  development  have  a  Poisson 
distribution  with  conditional  expected  value  E  D{b  +  \\M(Td(b  +  \)y,Td(b  +  \)^\KQ 


K0  is  introduced  to  represent  a  combined  deterministic  and  random  component  of 
between-copy  variability  in  Block  b+ 1.  The  Poisson  is  an  acceptable  distribution. 


DT  and  OT  are  pooled  in  this  model.  Each  remaining  DD  is  activated  during  a  test 
with  probability  1  -6-6  .  Several  DDs  may  be  activated  during  a  test.  Any  DDs 
activated  during  a  test  are  presumed  to  be  removed  (an  optimistic 
simplifying  assumption). 


The  testing  policy  is  to  allocate  zTd  (b  + 1)  DT/OT  tests:  The  parameter  r  is  an 


average  number  of  DT/OT  tests  conducted  per  spiral  time  unit.  Each  test  uses  one 
copy  of  Block  b+\ .  The  conditional  distribution  of  the  number  of  DDs  remaining  after 
development  and  testing,  given  K0 ,  has  a  distribution  with  mean 


E 


D(b  +  \-M(Td{b  +  \))-Td(b  +  \)-r)\KQ 


DDs  remaining  in  Block  b+ 1  after  fielding  have  the  potential  of  causing  mission 
failure  if  activated  during  a  mission.  Let  0F  be  the  probability  with  which  a  DD 
remaining  in  Block  b+ 1  after  fielding  does  not  activate  during  a  mission.  The 
conditional  probability  of  mission  success  for  Block  b+ 1,  given  completion  of 
development  and  testing,  and  given  the  number  of  remaining  DDs,  is 

PF  (b  + 1 1  D(b  + 1  ;M(Td  (b  + 1  ));Td  ( b  + 1);  r)) 

(2) 

-  *  (*♦  i ;  Ji  (»  ♦  i );«  to  (»  +  ' 

All  fielded  copies  of  Block  b+ 1  have  the  same  remaining  DDs  since  all  are 
constructed  by  the  currently  evolved  design.  This  is  a  oversimplification  that  can 
be  relaxed. 


2,1  Parametric  Examples 

a)  Let  K0-k,  a  constant  and  the  conditional  distribution  of  DDs  introduced  during 
development  is  Poisson.  Then  the  probability  that,  after  fielding,  no  remaining  DDs 
activate  during  a  mission  is 


E 


D{b+\-M{Td(b+\))jd{b+\)-r) 

UF 


■  exp 


-V[m(Td(b  +  \))yT^b+X\\-9F) 


where  /[ L]  is  an  (ultimately)  increasing  function  of  m  (□) . 


(3) 
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b)  Assume  K0  is  random  with  gamma  distribution  of  scale  v  >  0  and  shape  parameter 


0  <  p ,  and  Laplace  transform 


E 


.  This  classical  modification  of  the 


l  J  LA  +  sJ 

Poisson  leads  to  the  Negative  Binomial.  The  expression  for  the  probability  of  no  DDs 
activating  during  a  field  mission  (field  mission  survival): 


E 


D{b+\-M(Td(b+\))jd(b+\)-T) 

UF 


I P 


V 

v+f[m{n)yT“(b+1\\-0F) 


(4) 


Further  randomization  is  possible  and  tractable  (e.g.,  randomization  of  J3  in  the  above 
exponent).  Such  a  step  can  be  interpreted  in  a  Bayes  manner  or  as  a  sensitivity  test. 


c)  Let  K0  be  random  obeying  a  positive  stable  law  with  scale  v  >  0  and  order 


0  <  p  <  1  and  Laplace  transform  E 


-  expj-(vs)^  j ;  see  Feller  II  (1966). 


Assume  K{)  is  the  minimum  of  the  stable  random  variable  and  a  truncating 
exponential  random  variable  having  mean  1/ /r;  in  this  case  the  Laplace  transform  of 


Kn 


is 


-sKn 


K 


P 


)and£[*o]  =  - 


1 


-{vkY 
-e  v  ' 


The 


probability  that  no  remaining  DDs  activate  during  a  mission  (field  mission  survival)  is 
given  explicitly;  see  Gaver  et  al.  (2004).  Auxiliary  randomization  as  in  (b)  and  (c) 
(dubbed  “double  stochasticity,”  c.f.  D.  R.  Cox  (1955),  M.  S.  Bartlett  (1955), 
M.  S.  Bartlett  (1967),  and  D.  R.  Cox  et  al.  (1980))  can  represent  environmental 
variation  during  a  mission  and/or  manufacturing  or  configuration  variability  between 
manufactured  copies.  Appropriate  parametric  statistical  estimation  to  evaluate  such 
expressions  is  an  important  problem  not  yet  addressed  by  the  authors. 


Note:  We  present  three  alternative  explicit  mathematical  fonnulas  for  probability  of 
field  mission  survival  by  Block  6+1.  These  are  all  directly  numerically  evaluated  on  a 
personal  computer,  or  even  a  handheld  calculator,  and  are  useful  for  checking  far 
more  elaborate  simulations. 


2.2  Acquisition  of  Block  6+1 

Let  B(b+ 1)  be  the  total  budget  to  develop,  test,  and  procure  Block  6+1.  Table  1  details 
(average)  cost  and  operational  parameters  required  at  this  stage. 
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Table  1:  Parameters 


Total  budget 

B(b  + 1) 

2,000 

Development  budget 

M 

variable 

Development  (spiral)  time 

Td(b  + 1) 

variable 

Rate  of  development  expenditure 

m=M/Td(b  + 1) 

variable 

Mean  number  of  tests  per  unit  time 

r 

variable 

Cost  Block  fo/copy/unit 

cm  (  b ) 

1 

Cost  Block  fo+l/copy/unit 

cm{b  + 1) 

1.2 

Cost  of  a  test 

ct 

1 

Cost  removing  each  DD  found 

during  test 

cr 

Mean  mission  arrival  rate 

A 

100 

Probability  mission  success  Block  b 

P(b ) 

0.4 

Utopian  requirement:  probability 
mission  success  Block  b+ 1 

P*{b  + 1) 

0.85 

ay 

0.5 

Parameters  of  effectiveness  growth 

a2 

2 

0.002 

0.8 

a2 

Parameter  for  introduction  of  DD 

«3 

2 

Rate  of  obsolescence  Block  b 

co(b) 

0.33 

Rate  of  obsolescence  Block  b+ 1 

co(b  + 1) 

0.2 

Probability  DD  survives  test 

9 

0.85 

Probability  DD  survives  mission 

9f 

0.8 

V 

2,000 

0.4 

Parameters  of  gamma:  scale,  shape 

P 

Here  is  a  breakdown  of  Block  b+ 1  budget  allocation: 


(5) 


B#(b  +  \)  =  B(b  +  \)-M(Td(b  +  l))-[cm(b  +  l)  +  ct]TTd(b  +  l) 

~crE[DR(b  +  \-Td(b  +  \)-r)\ 

where  DR  (b  +  \\Td  (h  +  l);r)  is  the  number  of  DDs  removed  by  testing.  After 
development  and  testing,  the  mean  number  of  copies  Block  b+ 1  fielded 
( b  +  1^ 

is(V^(h  +  l)  = — ) - b .  Each  of  these  copies  has  the  same  remaining  DDs  in  this 

c,n(b  +  \) 

initial  model,  and  hence  has  the  same  probability  of  field  reliability/suitability. 


2.3  Obsolescence  of  Block  b  and  Block  b+ 1 


Operational  obsolescence  is  inevitable,  and  is  a  powerful  reason  for  Block  upgrading. 
Assume  that  Block  b  will  become  obsolete  after  an  independent  exponential  time 
having  mean  \l  co(b) .  An  important  problem  not  addressed  yet  explicitly  is  to  develop 
an  adaptive  inferential  policy  for  one’s  own  force  (B)  to  conclude  that  the  opposing 


7 


force  (R)  has  found  a  countermeasure  to  B’s  asset,  such  could  take  the  form  of  first 
occurrence  of  a  “failure  run  of  r  (e.g.,  5)  B  mission  failures.  Such  would  signal  a 
needed  change  in  B’s  CONOPS,  or  a  needed  block  upgrade,  or  both.  After 
obsolescence,  the  probability  of  mission  success  for  Block  b  is  pQ(b)<  p(b ) ;  Po{b) 
may  be  close  to  zero.  Assume  a  fielded  Block  b+ 1  can  also  become  obsolete  after  an 
independent  exponential  time  having  mean  \/co(b  +  \).  After  obsolescence,  the 

probability  of  mission  success  for  Block  b+ 1  is  p0(b  +  \}<  p(b  +  \y  If  Block  b 

becomes  obsolete  before  the  completion  of  development  and  testing  for  Block  b+ 1, 
then  a  decision  can  be  made  to  continue  to  field  Block  b,  or  to  field  the  current 
Block  b+ 1  prematurely.  We  defer  discussion  of  this  possibility  to  Gaver  et  al.  (2005). 
In  this  paper,  we  will  assume  that  Block  b  is  completely  ineffective  after 
obsolescence;  that  is,  p0(b)  =  0;  we  also  assume  that  Block  b+ 1  is  completely 
ineffective  after  obsolescence. 


2.4  A  Basic  Decision  Problem 

Suppose  for  the  present  that  Block  b  does  not  become  obsolete  before  the  end  of 
development  and  testing  of  Block  b+ 1.  The  decision  to  be  made  is  whether  to  use  the 
budget  remaining  after  development  and  testing  of  Block  b+ 1  to  purchase  copies  of 
Block  b+ 1,  or  to  use  it  to  purchase  more  copies  of  Block  b.  Let  each  mission  use  one 
copy  of  a  Block  design.  Each  unit  of  a  Block  can  be  used  once.  DDs  remaining  in 
Block  b+ 1  after  fielding  can  cause  mission  failure.  The  decision  criterion  is  to 
maximize  the  expected  number  of  successful  missions.  The  decision  is  to  allocate  that 
amount  of  the  budget,  B(b+ 1),  to  upgrade  Block  b+ 1  so  the  expected  number  of 
successful  missions  by  the  copies  of  Block  b+ 1  procured  with  the  remaining  budget 
is  maximized. 


Assume  missions  arrive  according  to  a  stationary  Poisson  process  having  rate  A .  Let 
Np  ( b )  be  the  number  of  copies  of  Block  b  that  can  be  purchased  using  B(b+ 1).  The 
expected  number  of  mission  successes,  Sb ,  if  Block  b  is  purchased  exclusively  is 


^  -|W(G 


A  +  co^b) 
x  i- 


CO 


(*) 


2 


NF(b)p(b) 
1  NF{b) 


A  +  co(b) 


-NF(b) 


A 


A  +  co(b) 


NF{b)~  1 


CO 


(b) 


A  +  co(b ) 


P{b) 


(6) 


A  similar  expression  occurs  for  the  conditional  expected  number  of  successful 
missions  given  the  remaining  number  of  DDs  if  NF(b  + 1)  copies  of  Block  b+ 1  are 
fielded.  Let 
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(7) 


E 


=  PF 


(b 


(Nf  (b  + 1))  I  D(b  + 1  ;M(Td  (b  + 1  ));Td  (b  + 1); r) 
+  \\D(b  +  l;M(Td(b  +  l));Td(b  +  \);T)) 


~Nf(/>+l)-l 

v 

h 

2 

co(b  + 1) 

k 

_[_ 

2 

NF(b+l) 

(h  +  ll 

2-i 

k= 0 

+  co  (b  + 1) 

?i  +  co  (b  + 1) 

At 

+  co(b  + 1) 

tv  J7  l  U  T  1  1 

where  as  before  the  distribution  of  D(b  +  Y,M(Td  [b  +  l));Td  (6  +  l);r)  can  be  Poisson 
(K0  =  k) ,  or  Truncated  Stable  Poisson,  or  Gamma-Poisson  (negative  binomial). 

2.5  Example 

The  parameters  for  the  numerical  examples  appear  in  Table  1.  The  total  budget  for 
development,  testing,  and  procurement  of  Block  b+ 1  is  $2,000K.  Development 
budgets  $100K,  $200K,  and  $400K  are  considered.  Each  of  these  budgets  can  be 
spent  over  an  integer  time  interval  length  Td  of  1  time  period,  or  5  time  periods.  The 
rate  of  DT/OT  testing  ranges  in  integer  values  from  1  to  100  tests  per  time  period.  The 
proportionality  constant,  K{) ,  has  a  gamma  distribution  with  mean  equal  to  0.0002 
and  shape  parameter  0.4.  Table  2  summarizes  the  gains  made  by  an  appropriate 
choice  of  time  for  development  and  testing.  If  Block  b+ 1  is  not  developed  and  units  of 
Block  b  are  bought  with  the  budget,  then  the  expected  number  of  mission  successes  is 
120  (so  developing  and  buying  Block  b+ 1  is  advantageous). 

Table  2:  Maximum  Expected  Number  of  Successes  for  Block  b+ 1  after  Development  and  Testing; 
[Maximizing  Number  of  Tests];  Total  Budget=$2,000K. 


M/T:  rate  of 
expenditure  of 
development  budget 

20=100/5 

40=200/5 

80=400/5 

100=100/1 

200=200/1 

400=400/1 

M/B:  fraction  of  total 
budget  used  for 
development 

0.2=400/2000 

365 

[40] 

299 

[58] 

0.1=200/2000 

385 

[351 

346 
r  54i 

0.05  =100/2000 

396 

f251 

372 

[461 
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Expected  #  Successes  for  Block  b  +1  versus  Number  of  Tests 
Total  Budget=$2000K 


Number  of  Tests 
Figure  1 


— ■—  D-time=5;  D-budget=100 
— x—  D-time=5;  D-budget=200 
— D-time=1;  D-budget=100 
•  D-time=5;  D-budget=400 
D-time=1;  D-budget=200 
— *—  D-time=1;  D-budget=400 


E[Prob.  Mission  Success]  for  Block  b  +1  versus  Number  of  Tests 
Utopian  Probability  of  Mission  Success=0.85 
Total  Budget=$2000K 


I  — D-time=5;  D-budget=100 
— ■—  D-time=5;  D-budget=200 
x  D-time=5;  D-budget=400 
— A—  D-time=1;  D-budget=100 
D-time=1;  D-budget=200 
- D-time=1;  D-budget=400 


Discussion :  If  Block  b+ 1  is  not  developed  and  the  development  budget  is  spent  to 
acquire  copies  of  Block  b,  the  expected  number  of  mission  successes  using  Block  b  is 
120.  Figure  1  displays  the  expected  number  of  mission  successes  for  Block  b+ 1  as  a 
function  of  the  number  of  tests  for  various  developmental  budgets  and  developmental 
times.  Figure  2  displays  the  expected  probability  of  mission  success  as  a  function  of 
the  number  of  tests  for  various  development  budgets  and  development  times.  As 
Table  2  illustrates,  perhaps  surprisingly,  larger  expected  number  of  mission  successes 
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and  smaller  number  of  tests  needed  to  achieve  the  maximum  are  associated  with  a 
smaller  development  budget  and  longer  development  times. 

Tables  3  and  4  display  the  expected  probabilities  of  mission  success  after 
development  and  testing  for  different  distributions  of  Kq.  The  numbers  of  tests 
performed  for  the  results  in  Tables  3  and  4  are  the  same  as  those  displayed  in  Table  2; 
that  is,  they  are  the  number  of  tests  for  Ko  having  a  gamma  distribution  with  shape 
parameter  0.4  and  mean  0.0002,  which  maximize  the  expected  number  of  mission 
successes  for  Block  b+ 1.  In  Table  3,  the  distributions  considered  are  ffo=0.0002;  K0 
having  a  gamma  distribution  with  shape  parameter  =0.4  and  mean  0.0002;  K0  having 
a  positive  stable  law  distribution  with  order  equal  to  the  shape  parameter  of  the 
gamma  and  scale  the  same  as  the  gamma;  K0  having  a  truncated  stable  law 
distribution  with  the  same  scale  and  order  as  the  stable  and  the  exponential  truncation 
rate,  k,  chosen  so  that  the  mean  is  equal  to  0.0002.  In  Table  4,  the  distributions  of  K0 
are  the  same  except  the  shape  of  the  gamma  and  the  order  of  the  stable  law  are  0. 1 . 


Table  3:  Expected  Probability  of  Mission  Success  for  Block  b+ 1  after  Development  and  Testing: 
Constant  Ao=0.0002;  (gamma  K0,  shape  parameter  0.4,  and  mean  0.0002);  {stable  K0  with  order 
(scale)  equal  to  shape  (scale)  of  gamma};  (truncated  stable  ^T0)- 


M/T :  rate  of 
expenditure  of 
development  budget 

20=100/5 

40=200/5 

80=400/5 

100=100/1 

200=200/1 

400=400/1 

M/B:  fraction  of  total 
budget  used  for 
development 

0.2=400/2000 

0.79 

[0.79] 

{0.75} 

(0.79) 

0.67 

[0.67] 

{0.62} 

(0.67) 

0.1=200/2000 

0.82 

[0.82] 

{0.78} 

(0.82) 

0.74 

[0.74] 

{0.70} 

(0.74) 

0.05  =100/2000 

0.83 

[0.83] 

{0.79} 

(0.83) 

0.78 

[0.78] 

{0.75} 

(0.78) 
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Table  4:  Expected  Probability  of  Mission  Success  for  Block  b+ 1  after  Development  and  Testing: 
Constant  fSr0=0.0002;  (gamma  K0,  shape  parameter  0.1,  and  mean  0.0002);  {stable  K0  with  order 
(scale)  equal  to  shape  (scale)  of  gamma};  (truncated  stable  /f0)- 


M/T :  rate  of 
expenditure  of 
development  budget 

20=100/5 

40=200/5 

80=400/5 

100=100/1 

200=200/1 

400=400/1 

M/B:  fraction  of  total 
budget  used  for 
development 

0.2=400/2000 

0.79 

[0.79] 

{0.45} 

(0.79) 

0.67 

[0.67] 

{0.37} 

(0.67) 

0.1=200/2000 

0.82 

[0.81] 

{0.48} 

(0.82) 

0.74 

[0.73] 

{0.43} 

(0.74) 

0.05  =100/2000 

0.83 

[0.83] 

{0.48} 

(0.83) 

0.78 

[0.78] 

{0.46} 

(0.78) 

Discussion ;  The  expected  probabilities  of  mission  success  are  nearly  equal  for  the 
moment  and  shape-matched  versions  of  constant,  gamma,  and  truncated  stable  Kq, 
except  for  the  stable  with  order  /?  small  (0.1).  This  behavior  may  be  due  to  the 
much-exaggerated  shape  of  the  pure  stable  law  with  these  shape  parameters;  there  can 
be  many  small  values  versus  a  few  very  large.  The  mean  number  of  DDs  introduced 
when  Ko  has  a  pure  (untruncated)  positive  stable  law  is  infinite  and  the  expected 
probability  of  mission  success  after  development  and  testing  is  much  less  than  the 
others;  the  smallest  expected  probabilities  of  mission  success  occur  for  the  stable  law 
of  order  0.1.  This  suggests  the  need  for  a  sequential  stopping  rule,  a  problem  under 
current  investigation. 

3  Conclusions  and  Future  Program 

The  problem  discussed  is  simple  and  generic,  and  widely  encountered  in  defense 
acquisition.  In  subsequent  work  we  propose  to  elaborate  on  the  above  conditions  and 
issues,  and  to  provide  operational  tools  to  guide  the  timing  of  evolutionary  cycles.  We 
will  consider  the  development  and  testing  of  systems  consisting  of  subsystems  in 
series  or  in  a  time-dependent,  random  pattern  elsewhere.  We  will  also  consider 
Test-Analyze-Fix-Test  (TAFT)  testing  policies  with  a  sequential  stopping  rule  based 
on  first  occurrence  of  a  run  of  successful  (no  DDs  activating)  operational  tests. 
Simulation  and  more  analysis  will  be  designed  to  assess  the  procedure’s  robustness. 
Application  to  Interim  Armored  Vehicle,  IAV  (STRYKER)  acquisition  and  preview 
testing  is  underway. 


The  preceding  initial  decision-aiding  model  illustrates  the  steps  necessary  to  build  a 
case  for  a  new  block,  and  the  timing  of  its  initiation.  The  decision  should  depend  on 
the  improvement  of  performance  (effectiveness)  achieved  in  the  process  of  TAFT  and 
consequent  improvement  of  reliability  (suitability).  More  elaborate  model  examples 
are  required  to  exhibit  the  system-wide  capability  goals  needed  for  more  complex 
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externally  supported,  mobile  systems  to  operate  in  groups  (e.g.,  brigades)  with 
acceptable  mission  success  probability. 
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Appendix 


The  specific  parametric  example  used  for  illustration  is 


Gd[b  +  \;M(Td(b  +  \)\,m(Td(b  +  \))} 


\-Q^\~axM(Td{b  +  \))t 


exp  -a2 


m(Td{b  +  l)) 


«2 


(8) 


for  at  >0,  ai  >0  for  1=1,2.  Of  course,  (8)  is  hypothetical  and  speculative,  and  subject  to 
replacement.  Note  that  the  overall  fraction  of  possible  success  Gd  achieved  decreases  as  the 
time-rate  of  expenditure,  m,  increases,  (the  “haste  makes  waste”  effect).  The  conditional 
expected  number  of  DDs  introduced  during  development  is 


E \D(b  + 1  ;M(Td  ( b  + 1  ));Td  ( b  + 1))  |  K0 

L  J  (9) 

=  KQm(Td{b  +  l)f; 

the  expected  number  of  DDs  introduced  is  an  increasing  function  of  the  rate  of 
expenditure  during  development,  m.  For  this  parametric  form 

E\D(b  +  \-M(Td(b  +  \)\,Td(b  +  \)-T)\K0 

L  J  (10) 

=  K0m(Td{b  +  l)f3  t?r7^+l); 


for  K0  having  the  gamma  distribution  of  (4) 


6 


D{b+\-M{Td(b+\))-Td(b+\)-T) 


-\P 


v  +  m(Td(b  +  \))ai  0TTd(-b+l\\-dF) 


(11) 
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